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ABSTRACT 

This study investigated how the scientific competencies advocated by the Programme for International Student 
Assessment (PISA) could be improved by using online argumentation. An online argumentation system served as an aid 
for argumentation instruction and activities among students during the experiment. Seventy-one 10th grade high school 
students took part in this study. A quasi-experimental design was adopted and qualitative and quantitative analyses were 
used. The results showed that using the online argumentation system to conduct argumentation instruction and activities 
could improve the students' PISA scientific competencies. The experimental group students outperformed their 
counterparts in terms of mean scientific competencies. Finally, this study proposed suggestions related to argumentation 
research and science instruction. 
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1. INTRODUCTION 

The Programme for International Student Assessment (PISA) sponsored by the Organization for Economic 
Co-operation and Development (OECD) measures students' competencies in reading literacy, mathematics 
literacy, and science literacy every 3 years. Each survey focuses on a single subject, with the other two as 
supplementary subjects. The objective of the PISA is to understand how 15-year-old students from the 
participating countries apply knowledge and abilities in their handling of daily issues before completing their 
compulsory education (Bybee, 2008; Bybee & MaCrae, 2011; Fensham, 2009; Lin, Hong, & Huang, 2012a; 
Olsen & Lie, 2011; Sadler & Zeidler, 2009). Not only does the PISA measure the students' understanding of 
school curricula, it also measures the important knowledge and abilities required in civic society for their 
future (Bybee, 2008; Bybee & McCrae, 2011; Lin et al., 2012a; Yeh & She, 2010). Nowadays, public debates 
about governmental policies on socio-scientific issues also rely on the scientific literacy of citizens to make 
informed decisions (Lin et al., 2012a). 

The PISA focuses on the students' competencies in applying their acquired knowledge in daily scenarios 
(Bybee, 2008). The ability to probe is emphasized in their scientific literacy, including interest, support, and 
responsibility (OECD, 2009). Likewise, Taiwan's nine-year integrated curriculum for the science and 
technology domain is similar to the principles of the PISA in terms of cultivating the required competencies 
in students. However, the global rankings of Taiwanese students in scientific literacy fell from 4th place in 
2006 (total score of 534) to 12th in 2009 (total score of 520) (TWPISA National Center, 2012a). The 
students' performance scores are related to complex factors such as teaching practice, school culture, and 
family values. Although stating that the international evaluation results such as those of the PISA are due to 
education performances is controversial (Wang & Lin, 2005), it is still feasible to implement relevant 
teaching strategies in the classroom for promoting such performance. The enhancement of PISA scientific 
competencies has become an important issue in the reforming of school curricula by Taiwan's science 
educators. 

Recent studies on the PISA have focused on the effects of demographic variables or transnational culture 
(e.g., Basl, 2011; Bybee & McCrae, 2011; Kjarnsli & Lie, 2004; Olsen, 2005; Olsen & Lie, 2011; Nentwig, 
Roennebeck, Schoeps, Rumann, & Carstensen, 2009). Few studies (e.g., Lavonen & Laaksonen, 2009) have 
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proposed relevant teaching strategies for improving students' PISA scientific competencies. Researchers need 
to provide more experimental data in terms of PISA scientific competencies in order to contribute to 
educational policies. Some researchers (e.g., Lee, 2009) have recommended the improvement of 
argumentation skills in light of the relationship between PISA scientific competencies and argumentation, 
whereby it is thought that students would cultivate their scientific competencies. In addition, the internet is an 
effective tool for conducting argumentation instruction and activities (Bell & Linn 2000; Clark & Sampson, 
2008; Lin, Hong, & Lawrenz, 2012b; Yeh & She, 2010) and new generation has been used to contact in the 
social networking service, such as Facebook. The present study considered the use of internet tools and 
argumentation as a medium to explore the improvement of students' scientific literacy. 

1.1 Scientific Literacy 

The OECD (2009) has illustrated the definition of scientific literacy and its framework for the PISA science 
assessment. In this framework, the term scientific literacy denotes an overarching competency comprising a 
set of specific scientific competencies. Scientific competencies include the abilities to mobilize cognitive and 
non-cognitive resources in any given context. These competencies are regulated by the individual's 
appreciation, interest, values, and actions relative to scientific matters, and are characterized as crossing 
through three interrelated aspects, including context, knowledge, and attitudes (as shown in Fig. 1). 



Figure 1. Framework for the PISA science assessment (OECD, 2009) 

Scientific competencies comprise a set of three specific constructs (OECD, 2009): 

1. Identifying scientific issues: This construct includes recognizing issues that are open to scientific 
investigation; identifying keywords to use in searching for scientific information; and recognizing 
the key features of a scientific investigation. 

2. Explaining phenomena scientifically: This construct includes application of scientific knowledge in 
a given situation; describing or interpreting phenomena scientifically and predicting changes; and 
identifying appropriate descriptions, explanations, and predictions. 

3. Using scientific evidence: This construct includes interpreting scientific evidence, making and 
communicating conclusions; identifying the assumptions, evidence, and reasoning behind 
conclusions; and reflecting on the societal implications of science and technological developments. 

In the dimension of "context" in Fig. 1, the assessment is built around life situations including five 
aspects: health, natural resources, environment, hazard, and frontiers of science and technology, with each 
context spanning personal, social, and global scales. The dimension of "knowledge" includes "knowledge of 
science" and "knowledge about science". The former refers to knowledge of the natural world while the latter 
entails knowledge of scientific inquiry. The dimension of "attitudes" includes interest in science, support for 
scientific inquiry, and motivation to act responsibly towards, for example, natural resources and 
environments. 
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1.2 Argumentation and Online Argumentation 

Argumentation in scientific learning can be defined as a process of connecting claims and data through 
justification or through the evaluation of knowledge claims in light of empirical or theoretical evidence 
(Clark & Sampson, 2009; Jimenex-Aleixandre & Erduran, 2008; Osborne, Erduran, & Simon, 2004). 
Arguments are the artifacts of argumentation and consist of either assertions or conclusions, including their 
justifications, reasons, or supporting factors (Zohar & Nemet, 2002). In science education, argumentation is 
seen as entailing three overlapping goals: making sense of the phenomenon under study (i.e., constructing 
claims and explanations), articulating those understandings (presenting arguments), and persuading others to 
adopt one’s ideas (critiquing and evaluating counter ideas while defending one’s own) (Berland & Reiser, 
2009,2011). 

Discourse is an essential component of argumentation, and social interaction plays a critical role in 
knowledge construction. Such social interaction forms an environment for discussion which allows 
individual thinking to move from implicit to explicit, and this can result in group reflection to reach a 
common consensus. However, students are often not able to propose evidence or reasons supporting their 
arguments (Nussbaum, 2002). Argumentation should be imparted to students through appropriate teaching, 
learning tasks, assisting, and other processes (Kuhn, 1991). It would achieve such teaching objectives more 
easily by complementing them with scaffolding (Bell & Linn, 2000; Nussbaum, 2002; Osborne et al., 2004). 

Argumentation activities require specific steps and thinking processes, of which and information 
technology can assist in building (Bell & Linn, 2000; Yeh & She, 2010). Computer tools can assist students 
in becoming researchers and reviewers, helping them to actively evaluate scientific thinking (Linn, 1998). 
Joiner and Jones (2003) believe that asynchronous computer-mediated communication is much more 
advantageous than face-to-face communication and is suitable to become the cultivation tool of 
argumentation skills. Lor example, the characteristic of asynchronous online discussion is the extension of 
answering time for students (Joiner & Jones, 2003). High level questions can stimulate more accurate and 
definite answers from students when given a more sufficient waiting time. Moreover, the context of 
argumentation requires equality amongst the participants, which cannot be easily established in traditional 
classrooms (Duschl & Osborne, 2002; Joiner & Jones, 2003). Online discussion is free from the traditional 
classroom constraints whereby the discussion is led by a few dominant students, and low-achieving students 
can express their own opinions asynchronously. Therefore, this study hoped to complement argumentation 
instruction and activities with the advantages of an online environment, and observe how this process might 
promote students’ scientific literacy. 

1.3 Argumentation and Fostering of Scientific Literacy 

Argumentation is the coordination of evidence and theory to support or refute an explanatory conclusion, 
model, or prediction (Clark & Sampson, 2009; Osborne et al., 2004). Such a process requires PISA scientific 
competencies in using scientific evidence and explaining phenomena scientifically. The competency in using 
scientific evidence involves accessing scientific information and producing arguments and conclusions based 
on scientific evidence. The competency in explaining phenomena scientifically includes describing, 
interpreting phenomena, and predicting changes, and may involve recognizing or identifying appropriate 
descriptions, explanations, and predictions (OECD, 2009). The above comparison shows the potential 
relationship between argumentation and PISA scientific competencies. Therefore, conducting argumentation 
instruction and activities may promote the students’ scientific literacy. 

The improvement of PISA scientific competencies also requires students to go through the scientific 
learning process in the scientific context. Previous science learning studies have viewed individual learning 
as a form of conceptual change (Posner, Strike, Hewson, & Gertzog, 1982), and relevant researchers (Dole & 
Sinatra, 1998; Nussbaum & Sinatra, 2003; Yeh & She, 2010) believe that argumentation can effectively 
promote students’ conceptual change. The process of argumentation allows students to compare and contrast 
arguments through in depth thinking (Dole & Sinatra, 1998), essentially triggering a cognitive conflict in an 
individual by giving him/her new information that varies from his/her prior knowledge. This is the first 
condition proposed by Posner et al. (1982) for conceptual change: the learner has to feel dissatisfaction on a 
current concept. A cognitive conflict triggered by in-depth thinking can then lead to learning and conceptual 
change (Nussbaum & Sinatra, 2003). 
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1.4 Research Question 

Based on the above literature discussion, this research used an online argumentation system to complement 
the argumentation instruction and discussion of PISA issues. This study investigated the improvement of 
PISA scientific competencies throughout these online argumentation processes. The research question was as 
follows: Were there any differences in scientific competencies between students who participated in online 
argumentation instruction and activities and those who did not? 


2. METHODOLOGY 

2.1 Participants 

The subjects of the PISA are primarily 15 -year-old students worldwide who have completed compulsory 
education. Therefore, this study selected 10th grade high school students (15 years old) from a senior high 
school situated on the edge of the Kaohsiung city center in southern Taiwan. This study was conducted 
during the students’ first summer vacation when they had just finished their junior high school education. A 
total of 79 students were chosen from two out of six classes. Due to eight students applying for leave during 
the experiment process, a total of 71 students participated fully in the research throughout, with 36 of them in 
the experimental group and 35 in the control group. These students had similar academic achievements, 
having being promoted to senior high school from junior high through the national entrance examinations. 

2.2 Experimental Design 

The study adopted a quasi-experimental design (as shown in Table 1). A PISA science assessment was 
conducted on both groups before the experiment. The students in the experimental group then went through 
argumentation instruction and activity using an online argumentation system for a total of 4 hours. After the 
experiment, both groups conducted the same PISA science assessment. 


Table 1 . Experimental design of this study 


Group 

Pre-test 

Treatment 

Post-test 

Experimental group 

O, 

X 

0 3 

Control group 

0 2 


0 4 


Note : O b 0 2 , 0 3 , and 0 4 were the PISA science assessments; X was the online argumentation 


2.3 The Online Argumentation System 

The argumentation system is shown in Fig. 2. The upper portion shows the argumentation topic, for which 
the teacher can set up two opposing viewpoints. The lower left portion is the column for the tree directory of 
students’ arguments titles. This directory provides a way for both the teacher and students to view the 
arguments of all the students in the class by means of hyperlinking to their saved location in a database. In 
addition, this column records the thought process of all the students, helping individuals to reflect on their 
thoughts. The lower right portion shows the argument of an individual student. Students are able to 
understand the perspectives of other students and achieve the objective of mutual interaction. 

As shown in Fig. 3, the interface for students to construct their arguments was designed based on the 
Toulmin (1958) Argument Pattern (TAP). It provides the scaffold to assist students in formulating their 
arguments. To ensure that students are not affected by specific terms of the TAP, Nussbaum (2002) 
suggested that changes to spoken language should be made. Students are given partial statements with 
textboxes next to them where they can type information to complete the TAP structure. Separate scaffolding 
provides prompt buttons marked with a question mark for the students, which prompt the students according 
to the missing or incorrect components of their previous arguments. If a student’s information did not 
adequately complete the statement, the button remains visible and indicates that the information was not 
adequately complete. Should the preceding argument encompass all five components, then the buttons would 
not appear to achieve the ’’fading out” strategy. 
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Figure 2. The system overview 


Figure 3. The scaffold for argumentation 


2.4 The PISA Science Assessment 

The released items of the PISA science assessment can be downloaded from the TWPISA National Center 
(TWPISA National Center, 2012b). These sample items were constructed or translated by experts and 
researchers from various fields in the past when Taiwan participated in the PISA. The most items are the 
same as the OECD released items, which can be downloaded from the PISA official website (OECD, 2006). 
Five teachers in the research team from the biological, life sciences, physics, chemistry, and earth sciences 
field were tasked to select items, the constructs of which were then reviewed and verified by two science 
educators. To achieve construct validity, the Rasch model (Rasch, 1960) was applied for analysis and the 
infit mean square error (MNSQ) of each item was calculated to be between 0.95 and 1.18. In terms of 
reliability, the item separation reliability of the whole test was 0.95. The validity and reliability were both 
within acceptable range. The answering formats of the assessment items included multiple choice questions, 
true or false questions, and short answer questions. There were a total of 15 items, with each item having a 
full score of 10, a partial score of 5, and a score of zero for a wrong answer. The full score for the entire 
assessment was thus 150. 

2.5 The Argumentation Instruction and Activities 

Students in the experimental group went through online argumentation instruction and activities. 
Argumentation instruction focused on how to use the argumentation system and the TAP for constructing 
individual arguments. During the instruction, the researcher tried to allow individual students to construct the 
arguments that best suited their logical thinking by following the teacher's argument example. The teaching 
strategy enabled students' learning to progress from simplicity to complexity and from single to diversity 
(Collins, Brown, & Newman, 1989) in terms of argument construction. 

The three lessons of argumentation instruction were conducted, in order, using a three components 
argumentation pattern (claim-data- warrant), a four components argumentation pattern (claim-data-warrant- 
backing), and a five components argumentation pattern (claim-data- warrant-backing-rebuttal). The templates 
used for student practice in argumentation instruction were based on examples in Toulmin's book (Toulmin, 
1958). For example, "All three sisters of Ming have red hairs (warrant). As Hua is the fourth sister of Ming 
(data), she might have red hair as well (claim). The supporting reason is that their family members have red 
hair genetically (backing). The exception is that Hua's hair turns white during aging (rebuttal)." These 
examples include deductive reasoning and inductive reasoning. The researcher explained the difference 
between these two forms of reasoning in the argumentation instruction. 

Students experienced difficulties mostly in responding PISA items of identifying scientific issues, 
especially when they are required to describe specifically about the treatment and dependent variable of a 
contextual investigation (Lin et al., 2012a). The final lesson of current study conducted argumentation 
activities based on two topics, including the scientific process encompassing experimental design and control 
variables. Both topics are related to "identifying scientific issues", to compensate PISA-related competencies 
for the lack of training in this area for argumentation instruction. During the argumentation, the teacher did 
not provide any correct answer, just guiding the activities instead. 
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3. RESULTS 

The tests of homogeneity conducted before ANCOVA obtained the following results for “identifying 
scientific issues”, “explaining phenomena scientifically”, “using scientific evidence”, and “overall score”, 
respectively: F ( i ; 67) = 02 (p=. 870>.05), F (lj 67 )=15 (p=.6 95>.05), F (lj 67 )=. 14 (p=. 704>.05), and F (1? 67) =. 30 
(p=.583>.05). These results, in which all the tests did not reach significance, matched the basic assumption of 
regression analyses. Therefore, ANCOVA was conducted as illustrated below. 

The ANCOVA results for the various scientific competencies are shown in Table 2. Under the construct 
of “identifying scientific issues”, the post-test difference between groups achieved significance (F(i ?68 )=7.40, 
p<. 01) with an effect size of 0.098. According to Cohen’s (1988) definition, the small, medium, and large 
effect sizes are 0.10, 0.25, and 0.40, respectively. This shows that the experimental treatment had a medium 
effect on the competency in “identifying scientific issues”. The comparison in Table 2 shows an adjusted 
mean post-test score of 26.85 for the experimental group, higher than the 21.81 obtained for the control 
group. 


Table 2. The ANCOVA of scientific competencies 


Competencies 

Group 

Pre-test 

Mean(SD) 

Post-test 

Mean(SD) 

Post-test 

Mean’(SE) 

F 

h 2 

Cohen’ 

/ 

Identifying scientific 
issues 

Experimental 

Control 

26.19(6.81) 

23.25(8.61) 

27.41(7.36) 

21.22(8.95) 

26.85(1.28) 

21.81(1.30) 

7.40** 

0.098 

0.330 

Explaining phenomena 
scientifically 

Experimental 

Control 

30.88(9.14) 

28.17(8.32) 

33.83(7.38) 

30.62(8.22) 

33.18(1.10) 

31.29(1.12) 

1.47 

0.020 

0.143 

Using scientific evidence 

Experimental 

Control 

55.69(5.49) 

53.57(9.59) 

56.52(6.74) 

51.85(8.91) 

55.92(1.09) 

52.47(1.11) 

4.84* 

0.066 " 

0.266 

Overall score 

Experimental 

Control 

112.50(12.96) 

105.00(20.51) 

117.92(14.01) 

103.14(18.91) 

116.04(2.40) 
105.06(2.44) 
v—rrr-r, 

10.00** 

0.128 ~ 

0.383 ' 


Note : *p<.05; **p<.01; 5D=standard deviation; AE=standard error; Cohen's/ =rp/(l-r| ) 


Under the construct of “explaining phenomena scientifically”, the post-test difference between groups did 
not achieve significance (F(i >68 )=1.47, p>. 05) with a small effect size of 0.143. The adjusted mean post-test 
score of the experimental group was 33.18, compared to 31.29 for the control group. 

Under the construct of “using scientific evidence”, the post-test difference between groups achieved 
significance (F( i ; 6 8)=4.84, p<. 05) with an medium effect size of 0.266. This shows that the experimental 
treatment had a medium effect on the competency in “using scientific evidence”. The comparison in Table 2 
shows the adjusted mean post-test score of 55.92 for the experimental group, which is higher than the 52.47 
obtained for the control group. 

Finally under “overall score”, the post-test difference between groups achieved significance (F {1 _ 
68 )= 10.00, /?<.01) with an effect size of 0.383. This showed that the experimental treatment had a medium to 
large effect on the overall scientific competencies. The comparison in Table 2 shows an adjusted mean post- 
test score of 116.04 for the experimental group, which is higher than the 105.06 obtained for the control 
group. 


4. DISCUSSIONS 

In the post-test, the mean overall scientific competencies of the experimental group was higher than that of 
control group, as is the case with the competencies in ’’using scientific evidence” and ’’identifying scientific 
issues”. However, the experimental group students did not outperform their counterparts in terms of 
competency in "explaining phenomena scientifically”. The nature of argumentation is to produce scientific 
evidence to support conclusions, and the process of argumentation instruction appeared to promote the 
abilities of students to raise scientific evidence, such as answering questions based on data given in a 
diagram. The argumentation topics in the current study were based upon identifying scientific issues, and the 
argumentation process appeared to promote the students’ abilities to identify scientific topics, such as 
understanding control variables and experimental designs. The above two forms of competencies do not 
require much basic background knowledge from the students as compared to the competency in "explaining 
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phenomena scientifically”. Nevertheless, some items of the "explaining phenomena scientifically” in PISA 
require students to have basic background knowledge. For example, there is a question under "Acid Rain" 
that states: "Where do these sulfur oxides and nitrogen oxides in the air come from?" The official answer 
from PISA is: "Any one answer referring to car exhausts, factory emissions, burning fossil fuels such as oil 
and coal, gases from volcanoes, or other similar things." The scoring of such questions still requires students 
to have background knowledge, in order to allow students to answer questions effectively by applying the 
competency in "explaining phenomena scientifically". 


5. CONCLUSION 

In summary, argumentation instruction comprising a specific process, online scaffolding assistance, and 
argumentation conflict scenarios may help to promote students’ PISA scientific competencies. Therefore, 
using the online environment to complement argumentation instruction and organizing argumentation 
activities focused on related topics is a potential direction to consider in the future for improving students’ 
scientific literacy. 
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